Object tracking method and apparatus using template matching

ABSTRACT

An object is detected in comparison between an input image from an image pick-up device having a zoom mechanism and a template image stored in a manner that a first image within a view field of the image pick-up device is stored as a first as the template image, a power of the zoom mechanism to be changed is recorded, a second image to be detected is picked-up from the image pick-up device. Then, a size of either one of the template image and the second image is changed on the bases of the changed power of the zoom mechanism, and a template matching is performed between the template image and the second image to detect the object. This process makes it possible to track an object within the view field.

CROSS-REFERENCE TO RELATED APPLICATION

The present invention is related to U.S. patent application Ser. No.09/592,996 filed Jun. 13, 2000 entitled “OBJECT TRACKING METHOD ANDOBJECT TRACKING APPARATUS”

BACKGROUND OF THE INVENTION

The present invention relates to a monitor apparatus using an imagepickup unit, or in particular to an object tracking method forautomatically detecting an object intruding into an imaging field orimage pickup field from a video signal inputted from the image pickupunit and automatically tracking the motion of the detected object and anobject tracking apparatus for automatically adjusting the imagingdirection (central direction of an image) in accordance with thedetected motion of the object.

A video monitor apparatus using an image pickup unit such as atelevision camera (referred to as a TV camera) has been widely used. Asone of the monitor systems using such video monitor apparatuses, amanned monitoring system may be referred in which system an intrudingobject such as a man or an automotive vehicle entering the monitor fieldis detected or tracked by a human monitor while watching the imagedisplayed on the monitor. However, apart from such a manned monitoringsystem, an automatic monitor system using a video monitor apparatus hasbeen in demand, in which an intruding object is automatically detectedfrom the image inputted from an image input unit such as a camera, themotion of the object is automatically tracked and a predeterminedannouncement or alarm action can be taken.

For realizing such an automatic monitor system, the first step is todetect an intruding object in the view field by a so-called subtractionmethod or the like. The subtraction method is executed to take the stepsof comparing the input image obtained by an image pickup unit with areference background image prepared in advance (i.e. an image notincluding the object to be detected), determine the brightness (orintensity) difference for each pixel, and detect an area with a largedifference value as an object. The part of the input image (referred toas a partial image) corresponding to the position of the intrudingobject detected in this way is registered as a template, so that aposition associated with the maximum degree of coincidence with thetemplate image is detected in the sequentially inputted images. Thismethod is widely known as the template matching, and is described indetail, for example, in the U.S. Pat. No. 6,208,033.

Ordinarily, in the case of tracking an object (for example, an objectdetected by the subtraction method) using the template matching, thepartial image at the position of the object detected by the matchingprocess is sequentially updated as a new template image because thetemplate matching follows the change of the posture of the object. Thisprocess will be now described with reference to FIGS. 1 to 3.

FIG. 1 is a diagram useful for explaining the flow of the process ofdetecting an object intruded into the view field by the subtractionmethod and registering the detected object as an initial template imageso that it may be used for the template matching.

In FIG. 1, numeral S01 designates an input image, numeral S02 areference background image, numeral S03 a difference image between theinput image S01 and the reference background image S02, numeral S04 abinarized image of the difference image S03, numeral S05 a subtractionprocessing unit, numeral S06 a binarization processing unit (Th),numeral S07 a man-like object in the input image S01, numeral S08 aman-like difference image in the difference image S03 corresponding tothe man-like object S07, and numeral S09 a man-like object (man-likebinarized image) in the binarized image S04 corresponding to theman-like difference image S08. Further, numeral S10 designates acircumscribed rectangle of the detected man-like object S09, numeral S11an extraction processing unit (CL) of extracting a specified area fromthe input image S01, numeral S12 an extracted image constructed of theextracted area (template image), and numeral S13 a template image.

In FIG. 1, first, the input image S01 of e.g. 320×240 pixels is inputtedfrom a camera E01 into an intruding object monitor apparatus E05. Then,in the subtraction processing unit S05, the brightness differencebetween the input image S01 and the reference background image S02prepared in advance is calculated for each pixel thereby to acquire thedifference image S03 in which the calculated difference of each pixel isassumed to be the brightness value thereof. At the same time, theman-like object S07 in the input image S01 appears in the differenceimage S03 as a man-like difference image S08.

Next, in the binarization processing unit S06, the brightness value ofeach pixel of the difference image S03 having the difference value lessthan a predetermined threshold (for example, 20) is set to “0”, whilethe brightness value of each pixel not less than the threshold is set to“255” (assuming that one pixel includes 8 bits in this specification)thereby to obtain the binarized image S04.

In the process, the man-like object S07 picked up in the input image S01is detected as a man-like object S09 in the binarized image S04 (Sketchof the object detecting process using the subtraction method).

In FIG. 1, furthermore, the circumscribed rectangle S10 of the man-likeobject S09 detected by the subtraction method is detected. Then, in theextraction processing unit S11, an area represented by the circumscribedrectangle S10 is extracted from the input image S01. The extracted imageis registered as the template image S13 in the extracted image S12(Sketch of the process of registering the initial template image).

FIGS. 2A and 2B are diagrams useful for explaining in an prior art theflow of the process of detecting where of the input image the intrudingobject registered in the template image using the template matching islocated. In FIG. 2A, numeral M01 designates the extracted image obtainedby the subtraction method (corresponding with the extracted image S12 inFIG. 1). In FIG. 2B, numeral M02 designates an input image obtained bythe image pickup unit E01 and numeral M03 a template image.

Numeral M05 designates one of the dotted areas in the input image M02and represents the position of the template image M03. Numeral M06designates one of the dotted areas in the input image M02 and representsa search area of the template matching. Further, dx and dy represent thewidths of the search area (in which dx represents the horizontaldirection and dy the vertical direction). The widths dx and dy are setaccording to the amount of the apparent motion (the motion on the image)of the object to be tracked. For example, the widths dx and dy may beset as dx=50 pix and dy=15 pix.

The template matching is a process of searching the portion of themaximum degree of coincidence with the template image M03 in the searcharea M06 of the input image M02. As this coincidence may be used anindex called the normalized correlation obtained from the followingequation (1).

$\begin{matrix}{{\gamma\left( {x,y,u,v} \right)} = \frac{\sum\limits_{i = 0}^{W - 1}\;{\sum\limits_{j = 0}^{H - 1}\;{\left\{ {{f\left( {{x + i},{y + j}} \right)} - \overset{\_}{f\left( {x,y} \right)}} \right\}\left\{ {{g\left( {{u + i},{v + j}} \right)} - \overset{\_}{g\left( {u,v} \right)}} \right\}}}}{\left. \sqrt{\sum\limits_{i = 0}^{W - 1}\;{\sum\limits_{j = 0}^{H - 1}\;\left\{ {{f\left( {{x + i},{y + j}} \right)} - \overset{\_}{f\left( {x,y} \right)}} \right.}} \right\}^{2}{\sqrt{\sum\limits_{i = 0}^{W - 1}\;{\sum\limits_{i = 0}^{H - 1}\;\left\{ {{g\left( {{u + i},{v + j}} \right)} - {g\left( {{u + i},{v + j}} \right)} - \overset{\_}{g\left( {u,v} \right)}} \right\}}}}^{2}}} & (1)\end{matrix}$

In the equation (1), f( ) designates the input image, go the templateimage, (x, y) the coordinates in the search area M06 of the input image(called the matching area), and (u, v) the coordinates in the upper left(uppermost left portion) of the template image M03, in which in allfigures the origin (0, 0) is located in the upper left (uppermost left)portion of the image. Further, W designates a width of the templateimage (horizontal length), and H a height of the template image(vertical length). Moreover, in the equation (1), {overscore (f( ))} and{overscore (g( ))} represent the average brightness value of the inputimage and the average brightness value of the template image,respectively, which are represented by the equations (2) and (3).

$\begin{matrix}{\overset{\_}{f\left( {x,y} \right)} = {\frac{1}{WH}{\sum\limits_{i = 0}^{W - 1}\;{\sum\limits_{j = 0}^{H - 1}\;{f\left( {{x + i},{y + j}} \right)}}}}} & (2) \\{\overset{\_}{g\left( {u,v} \right)} = {\frac{1}{WH}{\sum\limits_{i = 0}^{W - 1}\;{\sum\limits_{j = 0}^{H - 1}\;{g\left( {{u + i},{v + j}} \right)}}}}} & (3)\end{matrix}$

The normalized correlation r(x, y, u, v) represents the degree ofcoincidence between the brightness value distribution of the area withthe width W and the height H, in which the position (x, y) of the inputimage f(x, y) is the upper left (uppermost left) coordinates (forexample, M05 in FIG. 2B), and the brightness value distribution of thearea with the width W and the height H, in which the position (u, v) ofthe template image g(u, v) is the upper left (uppermost left)coordinates (for example, M03 in FIG. 2A). In a case where thebrightness value of each pixel of the input image f( ) is equal to thatof the corresponding each pixel of the template image g( ), thenormalized correlation is 1.0. The template matching is a process ofdetecting the portion of the maximum coincidence with the template imageg( ) in the input image f( ).

That is, with the position (u, v) of the template image go as areference position, as the position (x, y) is being changed in the rangeof u−dx≦x<u+dx, v−dy≦y<v+dy, the corresponding position (x, y) to themaximum normalized correlation r(x, y, u, v) represented by the equation(1) is searched.

Apart from the normalized correlation, as the degree of coincidence maybe used the average absolute value of the difference of the brightnessvalue of each corresponding pixel between the area with the width W andthe height H in which the position (x, y) of the input image f( ) is theupper left (uppermost left) coordinates and the area with the width Wand the height H in which the position (u, v) of the template image g( )is the upper left coordinates.

In this instance, in a case where the brightness value of each pixel ofthe input image f( ) is equal to that of each corresponding pixel of thetemplate image g( ) the average absolute value is made to be zero (0).As the difference of the brightness value of each corresponding pixelbetween the input image f( ) and the template image go is made larger,the average absolute value is made larger, (which means the degree ofcoincidence is made lower).

In the instance of FIG. 2B, the man-like object to be tracked is at theposition of M04. The template matching process is executed to detect thearea M07. Since this area M07 is detected at the position where theman-like object on the template image M03 coincides with the man-likeobject on the input image M02, the video monitor apparatus determinesthat the man-like object to be detected is inside the area M07. That is,it is understood that the intruding object is moving from the area M05to the area M07. In this case, the movement of the intruding object maybe represented by an arrow M08 connecting the center of the area M05with the center of the area M07.

In turn, with reference to FIG. 3, the description will be oriented tothe exemplary process of tracking an intruding object within a viewfield by applying the template matching described with reference to FIG.2 into the sequentially inputted images. FIG. 3 is a diagram useful forexplaining the process of tracking an object by sequentially performingthe conventional template matching method.

In FIG. 3, numerals T01 a, T02 a, T03 a and T04 a designate extractedimages at time points t0−1, t0, t0+1 and t0+2, respectively. NumeralsT01 c, T02 c, T03 c and T04 c designate template images at time pointst0−1, t0, t0+1 and t0+2, respectively. Numerals T01 b, T02 b, T03 b andT04 b designate input images at time points t0, t0+1, t0+2 and t0+3,respectively, numeral T05 a template matching processing unit (MT), T06a template image updating unit (UD). In this figure, the time point t0+nmeans that the time is apart from the time point t0 by n frames, inwhich the processing time of one frame is 100 msec, for example.

The matching processing unit T05 compares the template image of theextracted image with the input image, detect the portion of the maximumdegree of coincidence with the template image in the input image, andobtain the positions T01 e, T02 e, T03 e and T04 e of the intrudingobject to be tracked at time points t0, t0+1, t0+2 and t0+3 (templatematching).

The template image update unit T05 specifies the image of the portion ofthe maximum degree of coincidence detected by the matching processingunit T04 as the position image of a new intruding object and replacesthe extracted image and the template image using the position image,thereby updating the template image.

Then, the description will be oriented to the template matching processand the template image updating process along time points t0−1, t0,t0+1, t0+2 and t0+3 with reference to FIG. 3.

First, the template matching process is executed by using the templateimage T01 c obtained at time point t0−1 and the input image T01 bobtained at time point t0. In the first processing frame, the templateimage T01 c is matched to the partial image S13 of the input image S01corresponding to the position of the circumscribed rectangle of theman-like object S09 detected by the subtraction method.

The template matching processing unit T05 detects the position image T01e of the intruding object T01 d by the template matching described withreference to FIGS. 2A and 2B.

The template image update unit T06 updates the extracted image from T01a to T02 a by using the input image T01 b having a new position image(template) T01 e as the extracted image and also updates the templateimage from T01 c to T02 c on the basis of the position image T01 e ofthe intruding object. By executing this kind of process at time pointst0, t0+1, t0+2 and t0+3 respectively, it is understood that theintruding object within a view field is moved in the sequence indicatedin the position images T01 e, T02 e, T03 e and T04 e.

SUMMARY OF THE INVENTION

The template matching is a process of detecting a portion of the maximumdegree of coincidence with the template image in the input image. Thus,it is necessary to make the intruding object on the template imagesubstantially equal to the intruding object to be detected in the inputimage. Hence, the change of a zooming ratio (zooming power) of the imagepickup unit makes the intruding object displayed on the template imagedifferent in size from the intruding object to be detected in the inputimage, which brings about a disadvantage that no precise positioning ofthe intruding object can be performed.

Further, whether or not the intruding object is detected is determinedon whether or not the maximum degree of coincide obtained by thetemplate matching is not less than a predetermined value. Thepredetermined value is 0.5, for example, in the case of using thenormalized correlation.

The change of a zooming ratio of the image pickup unit makes theintruding object displayed on the template image different in size fromthe intruding object to be detected on the input image, thereby causingthe degree of coincidence to be lower. As a disadvantage, therefore, ifthe degree of coincidence is not more than the predetermined value, thedisability to detect the intruding object is determined.

As described above, if the zooming ratio (zooming power) of the zoomlens undergoes a change, as a disadvantage, the object tracing methodusing the foregoing template matching of the prior art provides nocapability of precisely detecting the position of the intruding objectand further detecting the intruding object itself.

It is an object of the present invention to provide an object tracingmethod and apparatus which are arranged to obviate the foregoingdisadvantages of the prior art.

It is another object of the present invention to provide an objecttracking method and apparatus which are arranged to precisely detect andtrack an object with high reliability.

In carrying out the foregoing objects, according to an aspect of theinvention, an object detecting method for detecting an object incomparison between an input image from an image pick-up device having azoom mechanism and a template image stored including the steps of:

storing a first image within a view field of the image pick-up device asthe template image;

recording a power of the zoom mechanism to be changed;

picking-up a second image to be detected from the image pick-up device;

changing a size of either one of the template image and the second imageon the bases of the changed power of the zoom mechanism; and

performing a template matching between the template image and the secondimage to detect the object.

The prior art using the template matching has a disadvantage that theadjust of the zoom lens may make the target intruding object differentin size from the template image and the precise matching therebetweencannot be guaranteed, thereby being unable to track the object withreliability.

However, as described above, according to the present invention, thezooming ratio is calculated with reference to the ratio of a focaldistance (focal length) at the time of obtaining the template image to afocal distance at the time of obtaining the input image and thereby thetemplate image is magnified according to the zooming ratio. This allowsa target intruding object to be precisely tracked as controlling oroperating the zoom lens.

The present invention, therefore, provides a capability of eliminatingthe conventional restriction of the zoom lens control of the imagepickup unit, reliably tracking the intruding object, and greatlyspreading the application range of the video monitor apparatus.

Other objects, features and advantages of the invention will becomeapparent from the following description of the embodiments of theinvention taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram useful for explaining the process of detecting anobject using the conventional subtraction method;

FIGS. 2A and 2B are diagrams useful for explaining the process ofdetecting an object using the conventional template matching method;

FIG. 3 is a diagram useful for explaining the process of tracking anobject by sequentially performing the conventional template matchingmethod;

FIG. 4 is a diagram useful for explaining the operation of the processof tracking an object according to an embodiment of the presentinvention;

FIG. 5 is a flowchart illustrating the process according to anembodiment of the present invention;

FIG. 6 is a block diagram showing an arrangement of an object trackingapparatus according to an embodiment of the present invention;

FIG. 7 is a flowchart illustrating a transformation of the process shownin FIG. 5;

FIG. 8 is a graph showing an exemplary operation in the case ofincreasing a focal distance of a zoom lens in the embodiment of thepresent invention;

FIG. 9 is a graph showing an exemplary operation in the case ofincreasing a focal distance of a zoom lens in the embodiment of thepresent invention;

FIG. 10 is a flowchart illustrating an object tracking process accordingto a second embodiment of the present invention; and

FIG. 11 is a flowchart illustrating an object tracking process accordingto a third embodiment of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

In order to obviate the disadvantages that the intruding object cannotbe precisely detected and that the degree of coincidence between thetemplate image and the input image becomes lower if the intruding objectdisplayed on the conventional template image is different in scale(size) from the intruding object to be detected on the input image, theobject tracking method according to the present invention is arranged toperform the template matching so that the intruding object displayed onthe template image is made to be equal in scale (size) to the intrudingobject on the input image by magnifying or reducing the template imageon the basis of the change of a focal distance (focal length) of a zoomlens.

That is, the object tracking method according to the invention isarranged to calculate a magnifying or reducing a power of the templateimage with reference to a ratio of a focal distance of the zoom lens ofthe image pickup unit at a time of obtaining the input image from whichthe registered template image was extracted to a focal distance of thezoom lens of the image pickup unit at a time of obtaining the inputimage that is subjected to the template matching and to track an objectwithin the range of a view field of the image pickup unit by performingthe template matching processing as magnifying or reducing the templateimage. This arrangement allows the object to be tracked as changing thezoom ratio of the zoom lens of the image pickup unit.

The arrangement of the video monitor apparatus according to anembodiment of the present invention will be described with reference toFIG. 6. FIG. 6 is a block diagram showing an arrangement of the objecttracking apparatus according to an embodiment of the present invention.Numeral E01 designates an image pickup unit (for example, TV camera),numeral E02 a zoom lens, numeral E03 an electric swivel (for example, acamera pan and tilt head), numeral E04 an operation unit, numeral E04 aa first button annexed with the operation unit E04, numeral E04 b asecond button annexed with the operation unit E04. In the monitorapparatus (image processing unit) E05, numeral E05 a designates an imageinput I/F (I/F: Interface), numeral E05 b a pan and tilt head controlI/F, numeral E05 c a lens control I/F, numeral 05 d an operation inputI/F, numeral E05 e an image memory, numeral E05 f an image output I/F,numeral E05 g an alarm output I/F, numeral E05 h a CPU (CentralProcessing Unit), numeral E05 i a program memory, numeral E05 j a workmemory, numeral E05 k a data bus, numeral E06 an output monitor, numeralE07 an alarm lamp, and numeral E05 an intruding object monitor apparatusarranged to have the image input I/F E05 a, the tilt and pan headcontrol I/F E05 b, the lens control I/F E05 c, the operation input I/FE05 d, the image memory E05 e, the image output I/F E05 f, the alarmoutput I/F E05 g, the CPU E05 h, the program memory E05I, the workmemory E05 j and the data bus E05 k. In addition, the image memory andthe work memory may be composed of a single memory.

In FIG. 6, the TV camera E01 is connected to the image input I/F E05 a,the zoom lens E02 is connected to the lens control I/F E05 c, the camerapan and tilt head E03 is connected to the pan and tilt head control I/FE05 b, the operation unit E04 is connected to the operation input I/FE05 d, the output monitor E06 is connected to the image output I/F E05f, and the alarm lamp E07 is connected to the alarm output I/F E05 g.Further, the image input I/F E05 a, the pan and tilt head control I/FE05 b, the lens control I/F E05 c, the input I/F E05 d, the image memoryE05 e, the image output I/F E05 f, the alarm output I/F E05 g, the CPUE05 h, the program memory E05I and the work memory E05 j are allconnected to the data bus E05 k.

In FIG. 6, the TV camera E01 is mounted on the camera tilt and pan headE03 and provides the zoom lens E02. The TV camera E01 picks up an imageof an object monitored (within the range of a monitor field), convertsthe picked-up signal into a video signal, and then output the videosignal into the image input I/F E05 a of the object tracking apparatusE05.

The image input I/F E05 a converts the picked-up video signal into theimage data of a type to be processed by the object tracking apparatusE05 and outputs the converted image data into the image memory E05 ethrough the data bus E05 k. The image memory E05 e accumulates theimages being inputted.

The CPU E05 h reads out the images accumulated in the image memory E05 ein accordance with the program saved in the program memory E05 i inadvance and then analyzes the read images in the work memory E05 j. TheCPU E05 h controls the zoom lens E02 through the lens control I/F E05 cfrom the data bus E05 k along the analyzed result or control the cameratilt and pan head E03 through the tilt and pan control I/F E05 b so asto change the view field of the TV camera E01, turn on the alarm lampE07 through the alarm output I/F E05 g, and display the image with anintruding object detected, for example, on the monitor E06 through theimage output I/F E05 f.

The image memory E05 e provides a template image holding unit for savingthe registered template images.

The following embodiment includes the hardware arrangement of the objecttracking apparatus described with reference to FIG. 6.

One embodiment of the invention will be described with reference to FIG.5. FIG. 5 is a flowchart illustrating the process of the embodiment.

First, in step 201 of an initial view field movement, the zoom lens E02and the camera tilt and pan head E05 are moved to a predeterminedinitial position so that the TV camera E01 may pick up an initial viewfield range (initial monitor field range). The initial view field is aview field within which displayed is a doorway of a specific building tobe specifically monitored, for example.

The following steps 202 to 206 represent the process from detection ofan intruding object to registration of a template image using thesubtraction method described with reference to FIG. 1. Hereafter, theprocess will be described with reference to FIG. 1.

First, in the step 202 of inputting an image, the input image S01 of 8bit/pix with a width of 320 pix and a height of 240 pix is obtained fromthe camera E01.

In the step 203 of subtraction, the operation is executed to calculate adifference of a brightness value of each pixel between the input imageS01 obtained at the step 202 of inputting an image and the referencebackground image S02 recorded in the image memory E05 e in advance andthen obtain the difference image S03 with the difference value as thebrightness value of the corresponding pixel.

Then, in the binarizing step 204, the operation is executed to compareeach pixel value of the difference image S03 obtained in the subtractionstep 203 with a threshold value and then obtain the pixel value of “0”if the pixel value is less than the threshold value (e.g., 20) or thepixel value of “255” if it is not less than the threshold value.

Next, in the object determining step 205, the operation is executed todetect a cluster S09 of the pixels each having a pixel value of “255” byusing the known labeling method and then to determine if an objectexists in the cluster of pixels. If yes, the process is branched intothe template image registration step 206, while if no, the process isbranched into the image input step 202, in which the next latest(present picked-up) input image is obtained, and then the process of thesteps 203 to 205 is executed.

In the template image registration step 206, the operation is executedto calculate the circumscribed rectangle S10 of the binarized object S09detected in the process of the image input step 202 to the objectdetermining step 205, cut the corresponding area to the circumscribedrectangle S10 from the input image S01, and then register the area asthe initial template image S13 in the image memory E05 e.

The steps 207 to 216 correspond to the process of tracking the intrudingobject S07 detected by the subtraction method through the use of thetemplate matching.

Like the image input step 202, in the image input step 207, the latest(present picked-up) input image (M02) of 8 bit/pix with a width of 320pix and a height of 240 pix, for example, is obtained from the TV cameraE01.

Then, in the zoom control determining step 208, in a case where the sizeof the template image of the intruding object being tracked is notfitted within the predetermined range, the zoom lens control isrequired. Hence, the operation is branched into the zoom lens controlstep 209. On the other hand, if it is fitted within the predeterminedrange, the operation is branched into the zooming ratio calculatingstep.

Herein, the predetermined range means such a range within which theentirety of the intruding object is contained within the view field ofthe image pick-up device. For example, the apparent height (length inthe vertical (y) direction: for example, H in FIG. 2B) of the intrudingobject is adjusted to be fitted within the range of ⅓ to ½ of the totalheight (for example, H0 in FIG. 2B) of the input image. This isperformed by using such a feature that an intruding object is seen to beelongated in the vertical direction and hence the entire image of theintruding object can be captured within the image from the TV camera E01so long as the entire image along the vertical direction of theintruding object is extracted and displayed within the screen. In thiscase, therefore, the apparent height of the intruding object is adjustedto be within the range of 80 to 120 pix.

In addition, the predetermined range is not limited to the foregoingrange. What is required is simply to adjust the distance from theposition of the intruding object into the vertical and horizontal edgesof the input screen to be not less than the apparent moving distance ofthe intruding object per one frame of the input screen. This is because,since the template matching is a process of detecting an image of anintruding object from an input image, if the intruding object moves outof the screen (that is, the input image) in the next processing frameafter updating a template, the detection of the intruding object usingthe template matching can not be performed. For example, therefore, ifthe intruding object is moved only in the horizontal direction, it isjust necessary to fit the apparent height of the intruding object intothe range of the total height of the input image.

Next, in the zoom lens control step 209, in order to suitably adjust thesize of the intruding object in the input image, if the size of thetemplate image is less than the predetermined range, the operation isexecuted to zoom in the zoom lens E02 (make the focal distance longer)through the lens control I/F E05 c. On the other hand, if the size ofthe template image is more than the predetermined range, the process isexecuted to zoom out the zoom lens E02 (make the focal distance shorter)through the lens control I/F E05 c.

This operation makes it possible to control the zoom lens E02 so thatthe apparent size of the intruding object may be fitted within thepredetermined range, that is, it may be the foregoing set value. Thistype of zoom lens control is realized by performing the feedback controlof the zoom lens E02 so as to be the set value on the basis of theapparent size of the intruding object.

Then, in the zooming ratio calculating step 210, the process is executedto calculate a ratio of a focal distance ft of the zoom lens at a timeof obtaining the extracted image M01 of the registered template imageM03 (at a time of inputting the template image registered in the step206, that is, at a time of inputting the template image or at a time ofinputting the template image updated in the step 215 to be discussedbelow, that is, at a time of inputting an image in the step 207) to afocal distance fi of the zoom lens at a time of obtaining the inputimage M02 and then calculate a zooming ratio (zooming power) based onthe ratio. Herein, one processing frame designates the intruding objecttracking process by using the template matching based on the input imagefrom the step 207 to the step 212.

The zooming ratio is 15/12=1.25 if ft=12 mm and fi=15 mm, for example.

Further, in the template magnification and reduction step 211, theoperation is executed to magnify (in which case the zooming ratio ismore than 1.0) or reduce (in which case the zooming ratio is less than1.0) the template image based on the zooming ratio obtained in thezooming ratio calculating step 210. In this process, for example, in acase where the size of the template image before magnification orreduction is horizontally 30 and vertically 50 and the zooming ratio is1.25, the size of the magnified or reduced template image is made to behorizontally 30×1.25=38 pix and vertically 50×1.25=63 pix (both of whichare rounded off).

Instead of magnifying or reducing the template image, the input imagemay be magnified or reduced. In this case, the zooming ratio becomes areciprocal number of the aforesaid zooming ratio, that is, ft/fi. In themethod of magnifying or reducing the template image, when the zoomingratio becomes smaller than 1.0, the number of the pixels constitutingthe template image becomes small, so that the reliability of thetemplate matching may be degraded. In this case, such a problem does notarise if the input image is magnified instead of reducing the size ofthe template. Although the magnifying or reducing processing of thetemplate image or the input image is performed electronically (on thepixel unit basis), the magnifying or reducing processing of the imagemay be performed by a zooming mechanism. For example, the magnifying orreducing processing of the image may be performed optically bycontrolling a zoom lens based on the calculated zoom ratio.

In the template matching step 212, the process is executed to search thepartial image with the maximum degree of coincidence with the templateimage in the input image and detect where the intruding object islocated by using the template matching described with reference to FIGS.2A and 2B.

Then, in the coincidence determining step 213, the process is executedto determine if the maximum degree of coincidence obtained by thetemplate matching in the step 212 is not less than the predeterminedvalue (for example, 0.5 if the normalized correlation is applied to thetemplate matching). If the maximum degree of coincidence is less thanthe predetermined value, it is determined that no intruding object isdetected (no corresponding partial image to the template image is foundin the input image). Hence, the process is branched into the initialview field moving step 201 from which the process of the steps 201 to212 is repeated. Further, if the maximum degree of coincidence is notless than a predetermined value, it is determined that any intrudingobject is detected. Hence, the process is branched into the camera tiltand pan head control step 214.

In the camera tilt and pan head control step 214, the camera tilt andpan head E03 is controlled on the basis of the position of the detectedintruding object. This control is executed to pan the camera tilt andpan head E03 to the right hand if the intruding object (the center ofthe partial image detected in the input image) is located in the righthand from the center of the image, pan it in the left hand if it islocated in the left hand from the center of the image, tilt upward if itis located upward from the center of the image, or tilt downward if itis located downward from the center of the image.

In the pan and tilt head control step 214, the pan and tilt motor of thecamera pan and tilt head E03 is controlled based on the displacementbetween the image center and the position of the target object(intruding object)detected by template matching, i.e. the direction ofthe target object with respect to the optical axis of the camera.Specifically, the center position of the target object detected bytemplate matching is compared with the center position of the image, andin the case where the center position of the target object detected islocated to the left of the center position of the image, the pan motorof the camera pan and tilt head is controlled to move the optical axisof the camera leftward, while in the case where the center position ofthe target objected is located to the right of the center position ofthe image, on the other hand, the pan motor of the camera pan and tilthead is controlled to move the optical axis of the camera rightward.Also, in the case where the center position of the target objectdetected is located above the center position of the image, the tiltmotor of the camera pan and tilt head is controlled to move the opticalaxis of the camera upward, while in the case where the center positionof the target object detected is located below the center position ofthe image, the tilt motor of the camera pan and tilt head is controlledto move the optical axis of the camera downward. The pan motor and thetilt motor can be controlled at the same time. In the case where thecenter position of the target object detected is located to the leftabove the center position of the image, for example, the tilt motor ofthe camera pan and tilt head is controlled to move the optical axis ofthe camera leftward while at the same time controlling the pan motor tomove the optical axis of the camera upward. By doing so, the camera panand tilt head can be controlled in such a manner as to hold the targetobject on the optical axis of the camera.

Further, in template updating step 215, the process is executed toperform the similar process to the template image update unit T06described with reference to FIG. 3. That is, the operation is executedto cut the corresponding area to the partial image M07 with the maximumdegree of coincidence with the template image from the input image M02and then update the template image with the cut area as a new templateimage M03.

In alarming and monitoring step 216, the process is executed to displayan image of a target object on the monitor E06 or light the alarm lampE07 through the alarm output I/F E05 g, for reporting the existence ofan intruding object.

In turn, the description will be oriented to the process of performingthe template matching as magnifying or reducing the template image byrepeating the process of the steps 207 to 216 for tracking an intrudingobject with reference to FIG. 4. FIG. 4 is a diagram useful forexplaining the process of tracking an object according to an embodimentof the present invention. As shown in FIG. 4, first, the process isexecuted to track an intruding object within the range of a view fieldof the image pickup unit by using the extracted image Z01 a based on theinput image obtained at time point T1−1 and the template image Z01 d.

Each of the images Z01 a, Z02 a and Z03 a is the extracted imageobtained on the input image at time point T1−1, T1 or T1+1. Further,each of the images Z01 d, Z02 d and Z03 d is the template image obtainedon the input image at time point T1−1, T1 or T1+1. Each of the imagesZ01 c, Z02 c and Z03 c is the input image at time point T1, T1+1 orT1+2.

The template image Z01 d based on the input image obtained at time pointT1−1 is processed as follows in a magnification and reduction unit (ZM)Z04 (steps 210 and 211). That is, the process is executed to calculate azooming ratio with reference to a ratio of a focal distance of a zoomlens at a time of obtaining the original input image of the extractedimage Z01 a from the TV camera to a focal distance of the zoom lens at atime of obtaining the input image Z01 c from the TV camera (step 210),magnify or reduce the template image Z01 d using the calculated zoomingratio, and then obtain the magnified or reduced extracted image Z01 band the template image Z01 e.

Herein, the process of increasing the focal distance of the zoom lenswill be described with reference to FIGS. 8 and 9.

FIG. 8 is a graph showing the increase of the focal distance of the zoomlens when zooming in the lens as a result of executing the presentinvention. FIG. 8 represents the increase of the focal distance of thezoom lens against the zoom lens control time t in controlling thezoom-in operation.

FIG. 9 is a view showing the increase of the focal distance of the zoomlens when zooming in the lens as a result of executing the presentinvention. FIG. 9 represents the decrease of the focal distance of thezoom lens against the zoom lens control time t when controlling thezoom-out time.

In FIG. 8, the zoom-in control is started at time point t=0, forexample.

In this case, if it is assumed that a delay of time td (0.1 sec, forexample) takes place until the zoom lens is started. Then, the zoom lensis accelerated for a time ta (0.5 sec, for example), the focal distanceis incremented by Δfmax and the zoom lens is stopped at time point ts, adelay of time td (0.1 sec, for example) takes place and then the zoomlens is decelerated for a time tb (0.5 sec, for example). This is theaccelerating and decelerating characteristic 701 in this case. Herein,the actual increase of the focal distance corresponds to a trapezoidarea of the accelerating and decelerating characteristic 701. Thezoom-in operation for a time of ts seconds allows the focal distance tobe increased by (2ts−ta+tb)×Δfmax/2 (when ts≧ta),(ta+tb)×ts²/ta²×Δfmax/2 (when ts<ta).

Assuming that the focal distance before the zoom-in operation is 10.0mm, the zoom-in operation of Δfmax=2.0 mm/sec and t=0.5 sec causes thefocal distance to be up to 20.0 mm and the zooming ratio of the inputimage before and after the zoom-in operation is 20.0/10.0=2.0. Like thezoom-out control in FIG. 8, the zoom-out operation for a time of ts seccauses the focal distance to be decreased by (2ts−ta+tb)×Δfmax/2 (whents≧ta), (ta+tb)×ts²/ta²×Δfmax/2 (when ts<ta).

In the magnification and reduction unit Z04, the extracted image Z01 band the template image Z01 e are magnified or reduced according to thiszooming ratio.

Then, in the template matching unit (TM) Z05 (step 212), by performingthe template matching for the input image Z01 c using the magnified orreduced template image Z01 e, it is possible to obtain the partial imageZ01 g of the input image Z01 c as an image of coincidence. Hence, it isdetermined that the intruding object in the input image Z01 c exists inZ01 f.

The magnification and reduction unit Z04, the template matching unit Z05and the template image update unit Z06 are programs for performing themagnification and reduction, the template matching and the templateimage update, all of which are saved in a program memory E05 i inadvance.

Next, in the template image update unit Z06 (step 215), the extractedimage and the template image are updated on the detected partial imageZ01 g of the input image Z01 c. That is, the new extracted image Z02 aand the new template image Z02 d are obtained.

Afterwards, the obtained partial image Z01 g is made to be the newtemplate image Z02 d. This new template image is subjected to themagnification and reduction processing. Then, the magnified and reducedtemplate image Z02 e and the new input image at time point t1+1 are usedfor the template matching. Then, the new extracted image Z023 a andtemplate image Z03 d are obtained on the detected partial image Z02 g ofthe input image Z02 c.

The sequential execution of the similar process causes the intrudingobject Z02 f to be obtained at time point t1+1 and the intruding objectZ03 f to be obtained at time point T1+2 as controlling the zoom lensE02. This process makes it possible to track the intruding object in theinput image.

As noted above, according to the embodiment of the invention, in thezoom control determining step 208 and in the zoom lens control step 209,the zoom lens is controlled on the size of the intruding object. Thezooming ratio is calculated in the zooming ratio calculating step 210and the template image is magnified or reduced on the basis of thezooming ratio in the template magnification and reduction step 210.Hence, this embodiment makes it possible to accurately track theintruding object without deviating the template image from the targetintruding object as operating the zoom lens and without lowering thedegree of coincidence in the template matching.

In this embodiment, the image is zoomed in or out in the steps 208 and209 and then is inputted into the step 202 or 207. Like thetransformation shown in FIG. 7, the process of the steps 208 and 209 maybe executed between the steps 214 and 215. This holds true to the secondand the third embodiment to be discussed below.

The second embodiment of the invention will be now described withreference to FIG. 2. FIG. 10 is a flowchart illustrating the process ofthe second embodiment of the present invention. The process shown inFIG. 10 is arranged to have a zoom operation step 208′ in place of thezoom control step 208 in the process described with reference to FIG. 5and a zoom lens control step 209′ in place of the zoom lens control step209 therein. Hence, the other steps 201 to 207 and 210 to 216 aresimilar to those of the first embodiment, so that the description aboutthose steps is left out.

When the process of the steps 201 to 207 is executed as described inFIG. 5, the process proceeds to a zoom operation determining step 208′.

In the zoom operation determining step 208′, it is determined whether ornot an operator handles the operation unit E04 on the basis of theoutput given through the operation input I/F E05 d. If the operatorhandles the operation unit E04, the process proceeds to the zoom lenscontrol step 209′. On the other hand, if the operator does not handlethe operation unit, the process proceeds to the zooming ratiocalculating step 210.

In the zoom lens control step 209′, if the operator zooms in the lensthrough the operation unit E04, the focal distance of the zoom lens E02is increased by a predetermined value (1 mm, for example). The zoom-inoperation is executed when the operator depresses the first button E04 bof the operation unit E04, for example. As an exemplary reference, eachtime the operator depresses the first button E04 a of the operation unitE04 once, the focal distance of the zoom lens E02 is increased by apredetermined value.

If the operator zooms out the lens through the operation unit E04, thefocal distance of the zoom lens E02 is decreased by a predeterminedvalue (1 mm, for example). The zoom-out operation is executed when theoperator depresses the second button E04 b of the operation unit E04,for example. As an exemplary reference, each time the operator depressesthe second button E04 b of the operation unit E04 once, the focaldistance of the zoom lens E02 may be decreased by a predetermined value.

If the operator zooms in and out the lens at a time, for example, thefocal distance of the zoom lens E02 may be kept constant. This makes itpossible for the operator to operate the zoom lens E02. Or, if theoperator zooms in and out the lens at a time, the operation unit E04 maybe arranged so that either one of the zoom-in and the zoom-outoperations may be executed.

Hereafter, the process after the step 210 is executed.

The process allows the intruding object Z02 f at time point T1+1, theintruding object Z03 f at time point T1+2 and the intruding object inthe input image to be tracked as handling the zoom lens E02.

As described above, according to the second embodiment of the presentinvention, in the zoom operation determining step 208′ and the zoom lenscontrol step 209′, the process is executed to control the zoom lens onthe basis of the operator's zooming operation. In the zooming ratiocalculating step 210, the zooming ratio is calculated. Then, in thetemplate magnification and reduction step 211, the template image ismagnified or reduced on the zooming ratio. Hence, since the operator canoperate the zoom lens as viewing the input screen displayed on themonitor E06, this process allows an intruding object to be accuratelytracked without deviating the template image from the target intrudingobject and without lowering the degree of coincidence in the templatematching.

In turn, the description will be oriented to the third embodiment of thepresent invention with reference to FIG. 11. FIG. 11 is a flowchartillustrating the process according to the embodiment of the presentinvention. The embodiment shown in FIG. 11 is a transformation of thesecond embodiment shown in FIG. 10. It is rearranged so that the zoomlens may be automatically controlled according to the size of theintruding object. That is, if the operator does not zoom in or out thelens, the zoom lens is automatically controlled so that the apparentsize of the intruding object may be displayed in a predetermined size.

FIG. 11 shows the second embodiment shown in FIG. 10 additionallyprovided with a zoom control determining step 301 and a zoom lenscontrol step 302.

The process of the steps 201 to 207, 208′ to 209′ and 210 to 215 are thesame as those of the foregoing second embodiment. Hence, the descriptionthereabout is left out.

In FIG. 11, the process of the zoom control determining step 301 isexecuted only when the operator does not zoom in or out the lens in thezoom operation determining step 208′.

In the zoom control determining step 301, if the size of the templateimage M03 of the intruding object being tracked is not fitted within theforegoing predetermined range, it is determined that the control of thezoom lens is required. Hence, the process is branched to the zoom lenscontrol step 302. On the other hand, if the size of the template imageM03 is fitted within the foregoing predetermined range, the process isbranched into the zooming ratio calculating step 210.

According to the third embodiment of the present invention, in the zoomoperation determining step 208′ and the zoom lens control step 209′, thezoom lens is controlled according to the operator's operation. If theoperator does not operate the zoom lens, in the zoom control determiningstep 302 and the zoom lens control step 302, the zoom lens iscontrolled. Then, the zooming ratio is calculated in the zooming ratiocalculating step 210. Next, in the template magnification and reductionstep 211, the template image is magnified or reduced according to thezooming ratio. Hence, as the operator is operating the zoom lens, thisembodiment allows an intruding object to be accurately traced withoutdeviating the template image from the target intruding object andwithout lowering the degree of coincidence in the template matching.

It should be further understood by those skilled in the art thatalthough the foregoing description has been made on embodiments of theinvention, the invention is not limited thereto and various changes andmodifications may be made without departing from the spirit of theinvention and the scope of the appended claims.

1. An object detecting method for detecting an object in comparisonbetween an input image from an image pick-up device having a zoommechanism and a template image stored in memory, said object detectingmethod comprising the step of: storing a first image within a view fieldof said image pick-up device as said template image; recording a powerof said zoom mechanism to be changed; pick-up a second image to bedetected from said image pick-up device; changing a size of either oneof said template image and said second image on the bases of saidchanged power of said zoom mechanism; and performing a template betweensaid template image and said second image to detect said object, whereinthe step of recording a power of said zoom mechanism to be changedcomprises the steps of: storing a first focal distance of said zoommechanism at a time when said template image is picked-up, and detectinga second focal distance of said zoom mechanism at a time when a secondimage from said image pick-up device is picked-up.
 2. An objectdetecting method according to claim 1, wherein said step of storing saidfirst image as said template image includes the steps of: setting anarea having a predetermined size containing said object and controllingsaid zoom mechanism so that said object is included in said area.
 3. Anobject detecting method according to claim 1, wherein said step ofperforming a template matching is a step of detecting a position of apartial image with a maximum degree of coincidence with said templateimage from said second image.
 4. An object detecting method according toclaim 1, wherein said zoom mechanism is either one of a zoom lens and anelectronic zoom.
 5. An object detecting method according to claim 4,wherein said zoom mechanism is said zoom lens, and said object detectingmethod further comprises the steps of: changing manually said focaldistances of said zoom lens and displaying said input image on a screenof a display, wherein an operator operates said zoom lens as viewingsaid input image on said screen.
 6. An object detecting method accordingto claim 4, wherein said zoom mechanism is said zoom lens, the focaldistance of said zoom lens is calculated after a passage of time from apredetermined reference time after a zoom control is started.
 7. Anobject detecting method according to claim 1, said object detectingmethod further comprises the step of: updating said template imagestored to a new template image on the bases of the results to perform atemplate matching.
 8. An object detecting apparatus for detecting anobject in comparison between an input image from an image pick-up devicehaving a zoom mechanism and a template image stored in memory, saidobject detecting apparatus comprising: a storing unit for storing afirst image within a view field of said image pick-up device as saidtemplate image; a recording unit for recording a power of said zoommechanism to be changed; a picking-up unit for detecting a second imagefrom said image pick-up device; a control unit for changing a size ofeither one of said template image and said second image on the bases ofsaid changed power of said zoom mechanism; and a processing unit forperforming a template matching, wherein said recording unit forrecording a power of said zoom mechanism to be changed stores a firstfocal distance of said zoom mechanism at a time when said template imageis picked-up and detects a second focal distance of said zoom mechanismat a time when a second image from said image pick-up device ispicked-up.
 9. An object detecting apparatus according to claim 8,wherein said control unit controls said zoom mechanism so that saidfirst image as said template image includes an area having apredetermined size containing said object and said object is included insaid area.
 10. An object detecting apparatus according to claim 8,wherein said zoom mechanism is either one of a zoom lens and anelectronic zoom.
 11. An object detecting apparatus according to claim10, wherein said zoom mechanism is said zoom lens, and said objectdetecting apparatus further comprises: means for changing manually saidfocal distance of said zoom lens and a display for displaying said inputimage, wherein an operator operates said zoom lens as viewing said inputimage on said display.
 12. An object tracking method for tracking anobject in comparison between an input image from an image pick-up devicehaving a zoom mechanism and a template image stored in memory saidobject detecting method comprising the steps of: storing a first imagewithin a view field of said image pick-up device as said template image;picking-up a second image to be detected from said image pick-up device;recording a power of said zoom mechanism to be changed; changing a sizeof either one of said template image and said second image on the basesof said changed power of said zoom mechanism; performing a templatematching between said template image and said second image to detectsaid object; and controlling a moving mechanism of said image pick-updevice in accordance with the results of said template matching so as totrack said object, wherein the step of recording a power of said zoommechanism to be changed comprises the steps of: storing a first focaldistance of said zoom mechanism at a time when said template image ispicked-up, and detecting a second focal distance of said zoom mechanismat a time when a second image from said image pick-up device ispicked-up.
 13. An object tracking method according to claim 12, whereinsaid step of storing said first image as said template image comprisesthe steps of: setting an area having a predetermined size containingsaid object and controlling said zoom mechanism so that said object isincluded in said area.
 14. An object tracking method according to claim12, wherein said step of performing a template matching is the step ofdetecting a position of a partial image with a maximum degree ofcoincidence with said template image from said second image from saidimage pick-up device.
 15. An object tracking method according to claim14, wherein said step of controlling a moving mechanism of said imagepick-up device further comprises the step of: controlling said movingmechanism so that said object to be tracked positions around of thecenter of said view field of said image pick-up device.
 16. An objecttracking method according to claim 12, wherein said zoom mechanism iseither one of a zoom lens and an electronic zoom.
 17. An object trackingmethod according to claim 16, wherein said zoom mechanism is said zoomlens, and said object detecting method further comprises the steps of:changing manually said focal distances of said zoom lens and displayingsaid input image on a screen of a display, wherein an operator operatessaid zoom lens as viewing said input image on said screen.
 18. An objecttracking method according to claim 16, wherein said zoom mechanism issaid zoom lens, the focal distance of said zoom lens is calculated aftera passage of time from a predetermined reference time after a zoomcontrol is started.
 19. An object tracking method according to claim 12,said object detecting method further comprises the step of: updatingsaid template image stored to a new template image on the bases of theresults to perform a template matching.
 20. An object tracking apparatusfor tracking an object in comparison between an input image from animage pick-up device having zoom mechanism and a template image storedin memory, said object detecting apparatus comprising: a storing unitfor staring a first image within a view field of said image pick-updevice as said template image; a picking-up unit for picking-up a secondimage to be detected from said image pick-up device; a recording unitfor recording a power of said zoom mechanism to be changed; a firstcontrol unit for changing a size of either one of said template imageand said second image on the bases of said changed power of said zoommechanism; a processing unit for performing a template matching betweensaid template image and said second image to detect said object; and asecond control unit for controlling a moving mechanism of said imagepick-up device in accordance with the results of said template matchingso as to track said object, wherein said recording unit for recording apower of said zoom mechanism to be changed stores a first focal distanceof said zoom mechanism at a time when said template image is picked-upand detects a second focal distance of said zoom mechanism at a timewhen a second image from said image pick-up device is picked-up.
 21. Anobject tracking apparatus according to claim 20, wherein said firstcontrol unit controls said zoom mechanism so that said first image assaid template image includes an area having a predetermined sizecontaining said object and said object is included in said area.
 22. Anobject detecting apparatus according to claim 20, wherein said zoommechanism is either one of a zoom lens and a an electronic zoom.
 23. Anobject detecting apparatus according to claim 22, wherein said zoommechanism is said zoom lens, and object detecting apparatus furthercomprises: means for changing manually said focal distances of said zoomlens and a display for displaying said input image, wherein an operatoroperates said zoom lens as viewing said input image on said display.